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Abstract 

We introduce a new method, stepwise method for solving optimal con¬ 
trol problems. Our first motivation for new approach emanate from limi¬ 
tations on continuous time control functions in PMP. Practically in most 
of the real world models, we are not able to change control value for every 
time such as in drug dose calculation or in resourse allocation problems. 
But it is practical to change control value in some time section that lead to 
stepwise control function. We study some examples via classical Pontrya- 
gin Maximum Principle(PMP) and via stepwise method. The new method 
has some other advantages in comparison with PMP method in models 
with complicated cost function or systems. In real world applications, the 
new method has a high performance in implementation. 


1 Introduction 

Optimal control theory is an effective tool in real world modelling such as physi¬ 
cal, biological, economical and other models. Diverse examples are studied in[T] 
and [2] . Optimal control theory is used in chemotherapy of cancer [3]. Several 
papers are studied about epidemiology from optimal control theory viewpoint, 
for example [4]. The most important classical method in optimal control the¬ 
ory is the remarkable result of Pontryagin Maximum Principle(PMP) that be 
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used in various forms in applied problems, there are necessary conditions in 
PMP that lead to limitations in applications. These conditions can occurred 
for functions in state equations, cost function and control functions. Here we 
interested in finding a new optimal control method with more proficiency in 
complicated cases. In this technique, control functions selected among the step¬ 
wise functions. Another notable point in the new manner is the combination 
of the heuristic and classical methods. In the forthcoming sections, numeri¬ 
cal forward-backward sweep method and stepwise method are applied to some 
problems such that one can obtain a clear vision about potency and power of 
the new method. Furthermore stepwise method works easily in the problems 
with more complicated cost functions in contrast to classical methods. In this 
paper the proficiency of stepwise method is shown in some models. 


2 Introductory example and definitions of step¬ 
wise method with fixed step-size 

In this section, we describe the stepwise method through the simple example. 
Consider the problem 

max{J = Jgi^x — 3it — u^)dt} 

subject to i: = a: -I- M, a;(0) = 5 and the control constrain m G H = [0, 2]. Solu¬ 
tion: We can find the optimal control function u{t) via PMP. The Hamiltonian 
is 


H = {2x — 3m — u^) + X{x -I- u) = (2 -I- X)x — (u^ + 3u — Xu). 

One can find the optimal control policy by differentiating H with respect to u. 
Thus 

l?=-2«-3 + A = 0, 

so that the control function is u(t) = that u(t) stays within the interval 

H = [0, 2]. We next drive the adjoint equation as 

A = -^ = -2-A, A(2)=0 

A-hA = -2, A(2)=0. 


This equation can be solved and X{t) = 2(e^ * “ 1) when we impose the control 
constraint H = [0, 2], the optimal control is obtained: 


u = 


2 if - 2.5 > 2, 

- 2.5 if 0 < e^-* - 2.5 < 2, 
0 if e^-* - 2.5 < 0. 


The final cost J is 68.93. For solving this problem by new method, we changed 
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Figure 1: Graph of optimal control for introductory example. 


the maximization problem to minimization through converting J to 1/(1 + J) 
and the final cost in minimization problem is 0.0143. When we use PMP, the 
control functions must satisfy the special conditions. In practical applications, 
we are not able to change control value every moment continuously. But we 
can change control values in some slices of time that lead to step function. In 
order to solve this problem by stepwise method, we seek control function among 
stepwise functions. For this purpose, we divide [0,T] into equal parts. Suppose 
that the control function u{t) has a constant value in each part. 

Let us enter the stepwise control function in control system x = f{x,u, 
t),x{0) = xo- For t € [0, ^],, we have u{t) = a and one can solve the ode 
X = f{x,a,t), a;(0) = xq- For t € we solve the ode x = f{x,l3,t) 

together with initial condition a;(0) = x{^) that is the terminal point of system 
in t G [0,^]. Same procedure repeat for x = f{x,"f,t), a;(0) = cc(^) when 

t G [^,T]. We can compute cost for a typical stepwise control function and 
search for the optimal values for (a,/3,7). In this manner we convert the op¬ 
timal control problem to optimization problem. For solving the optimization 
problems, we are able to use some analytical and heuristic and metaheuristic 
methods such as pattern search, simulated annealing, genetic algorithm, and 
other methods which these methods works easily in the problems with compli¬ 
cated systems and cost functions. There may be exist an important question 
here about the final cost. Is it possible that the difference between final cost in 
PMP method and in stepwise method exceeds from our expectation There is a 
simple lemma about stepwise functions and continuous functions can be helpful 
here. 

Lemma 2.1. For every continuous function u(t), there is a sequence {u„(t)} 
of stepwise functions that lim„^oo Unit) = u{t). 
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Using this lemma, we can be confident that the new method do not generate 
useless solutions. Note that some step functions are not able to satisfy the PMP 
condition and do not belong to admissible controls. But it is likely, we can find 
step function in new method with lower cost than PMP solution. The final 
cost in the stepwise method equal to 0.014305429952056222. We illustrate this 
method in figure below. Note that ,We have obtained all the numerical results in 
20-30 run times. We continue our study on 3-step stepwise method via pattern 
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Figure 2: Graph of optimal control for introductory example via stepwise 
method. 


search, simulated annealing and genetic algorithm. The results is illustrated in 
next figure and table. 


Method 

Final cost (J) 

Pattern search 
Simulated annealing 
Genetic algorithm 

0.014305429952056222 

0.014306824417196181 

0.014358890512889641 


Trying to get better stepwise control function through additional steps seems 
natural. We applied 5-step function instead 3-step function.These results coin¬ 
cide with our expectations. 


Method 

Final cost (J) 

Pattern search 
Simulated annealing 
Genetic algorithm 

0.014283994290191705 

0.01430531425509952 

0.0143199869929726 
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Figure 3: Results for stepwise method via pattern search, simulated annealing 
and genetic algorithm. 
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Figure 4: Results for 5-step stepwise method via pattern search, simulated 
annealing and genetic algorithm. 
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3 Stepwise method with variable step-size 

In the former section, we divided the interval into some equal parts. Here we let 
the optimization method decide about width of subintervals. Let’s come back 
to introductory example. Here, instead of dividing [0,2] to [0,1/3],[1/3, 2/3] 
and [2/3,2], we divide [0,a],[a, &] and [&, 2] and let the optimization method 
decide about a and b. The next figures and tables show the improvement in 
optimal policy and final cost and give complete information about subinterval 
and control value on the subinterval in variable stepwise method. 


Method 

Final cost (J) 

Pattern search 
Simulated annealing 
Genetic algorithm 

0.012566299700335069 

0.01334206413715155 

0.012912036300546229 


Method 

Subintervals 

control value 

Pattern search 
Simulated annealing 
Genetic algorithm 

[0,0],[0,1],[1,2] 

[0,0.0036],[0.0036,0.9738],[0.9738, 2] 
[0,0.0034], [0.0034,0.9027], [0.9027,2] 

(0,2,0) 

(1.6336,1.8345,0.5623) 

(0.7718,1.9087,0.1524) 



T]me(years, months, days, hours,...) 


Figure 5: Results for variable 3-step stepwise method via pattern search, simu¬ 
lated annealing and genetic algorithm. 


4 Stepwise method in real world models 

As we mentioned before, there are limitations for admissible controls in PMP 
approach such as continuity with respect to time and others. Practically, we 
are not able to change the value of control function in every moment of time 
interval. Instead, one can change the control value at several time sections.Thus, 
it seems that the stepwise method is a reasonable way in real world applications. 
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For example, when you want to make a decision about resource allocation in 
epidemiological models, you can not alter your strategy in short time interval. 
Because changing the vaccination rate or prevention strategy may imposes heavy 
costs. There are same problems in optimal control model of treatment of disease 
through the use of drugs. It seems that there are sufficient motivation to practice 
the stepwise method in real world processes. The next examples show the 
performance of stepwise method in contrast to classical PMP method. 

4.1 Example: Chemotherapy 

Optimal control methods are useful in optimal control model of chemotherapy. 
Renee fister et al in[3] studied different cell-kill models of chemotherapy. They 
characterized optimal control strategy that minimizes the cancer mass and the 
cost of total amount of drug. We use stepwise method for some of their models. 
The problem is: 



subject to, 


N'{t) = rNlnij^) - u{t)SN{t) 
7V(0) = No, u{t) > 0. 


The following parameters appear in model: 

N(t)\ The normalized density of the tumor at time t 
r: The growth rate of the tumor 
S: The magnitude of the dose 

u(t): The time dependent pharmacokinetics of the drug 
N^: The desired tumor density. 

Let us enter these value: r = 0.1, a = 3, b = 1, S = 0.45, Nd = 0, Nq = 0.975 
and T = 20. The next figures show the optimal control strategy in PMP and 
stepwise method. Now, we try to use stepwise method in this model and present 
the results below. The final cost in PMP method equal to 10.7758 and in 5-step 
stepwise method is 10.866632710096287. 

4.2 Example: Differential susceptibility and differential 
infectivity model 

Based on [4] , we develop optimal control formulation of DSDI model with two 
groups of susceptible and two groups of infected individuals [S]. Because of 
apparent diversity of examples, the idea of dividing susceptible and infected 
population into two subgroups examined. For example in plenty of diseases, 
disease processes is different in male or female, children or adult, adicted or 
nonaddicted, and so on. The following parameters appear in our model: 

/r: natural death rate; 

vf. the rate at which infectives in li are removed or become immune; 

6\ disease-induced mortality rates for the infectives; 

\i: The rate of infection for susceptibles in group Si; 
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Figure 6: Graph of optimal control for chemotherapy example via PMP method. 



Figure 7: Graph of optimal control for introductory example. 
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The infectivity rate Xi is given by Xi = rai J2j=i which ' 15[ is the trans¬ 

mission probability per contact and 'r' is the number of contacts of an individual 
per unit time. We suggested the following ODEs system 63) describing the 
model with controls. 

'^1 =/r(pi50-5i)-Ai5i(l-Mi) 

52 = KP2S° - S 2 ) - X2S2{1 - U2) 

' A = 9iiAi5i( 1 — Ml)-I-(72 iA252(1 — U 2 ) — (m + ^1 + ^ 3 ).^! (4-1) 

A = 912 Ai 5 i (1 — Ml) -I- g22A252(l — M2) — {11 + 122 + Ui)l 2 

R = {vi+ M 3 )/i -I- {V 2 + Ui)l 2 - (/i + 5 )R 

The control functions Mi(f), U2{t), u^{t) and M4(t) have to be bounded on [0,1] 
and Lebesgue integrable functions. Mi(t) and U2(t) measure the time dependent 
efforts on the preventive strategy of susceptible individuals in S, to reduce the 
number of individuals that may be infectious. The control functions U2(t) and 
M3 (t) measures the time dependent efforts on the treatment of infected individ¬ 
uals in Ii and I2 respectively. This control will have an impact on the output 
flow of people from the The objective functional to be minimized is: 

J(mi,M2,M3,M4) = f All + BI2 +Cu\+ Du\ + Eu\ + Fuldt ( 4 - 2 ) 
Jo 

Here, A, B, C, D, E are adjustment parameters. We seek an optimal control 
triple (mi,M2,m|,m|) such that 

J(m*,M 2,M3,M4) = min{J(Mi,M2,M3,M4)|(Mi,M2,M3,M4) G 17} 

where U = {J(mi, M 2 , 113 , U4)|Mi measerable, 0 <Ui < 1, t G [0, T], i = 1, 2, 3, 4} is the 
control set. Let us enter the following values in model system. 


Parameters and values 


S'^ = 1 

0 

II 

^ = .012 

51 ( 0 ) = 0.47 

T = 1000 

S'aA) = 0.47 

pi = 0.5 

/i(0) = 0.02 

P2 = 0.5 

/ 2 A) = -04 

Qi = 0.05 

R{0) = 0 

ct2 = 0.2 

di = 0.2 

Ml = 0.15 

d2 = 0.06 

M2 = 0.6 

r = 25 

gii = 0.9 

gi2 = 0.1 

921 = .1 

922 = .9 

T = 3 

B = 3 

C = 0.002 

D = 0.002 

E = 0.002 

F = 0.002 


Below, we can depict optimal control policy by PMP method and stepwise 
method(with pattern search for optimization problem).The final cost in PMP 
method equal to 0.1059 and in 5-step stepwise method is 0.11107136532373643. 
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Figure 8: Graph of optimal control for DSDI model via PMP. 




Figure 9: Graph of optimal control DSDI model via 3-step stepwise method. 
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5 conclusion 


we introduce the stepwise method for optimal control problems. This method 
could be replaced with PMP classic method in real world problems. Using this 
new method in various cases of applied models seems reasonable . 
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